Overview

This PR updates all OpenTelemetry and observability-related Helm charts to their latest stable versions and implements SHA pinning for enhanced security and reproducibility.

Motivation

The repository was using outdated versions of critical observability components:

• OpenTelemetry Collector was at version 0.108.0 (latest: 0.136.1)
• Prometheus was at version 25.4.0 (latest: 27.39.0)
• Grafana was at version 7.0.2 (latest: 10.0.0)
• Tempo was at version 1.7.0 (latest: 1.23.3)
• Loki was at version 5.36.3 (latest: 6.42.0)

Additionally, the charts lacked proper version pinning mechanisms to ensure reproducible deployments and protect against supply chain attacks.

Changes

Chart Version Updates

Security Enhancements

1. Cosign Signature Verification

All Helm charts now include cosign verification to ensure chart integrity:

chart:
  spec:
    verify:
      provider: cosign

This instructs Flux to verify cryptographic signatures before deploying any chart, protecting against tampered or malicious charts.

2. Container Image SHA256 Pinning

The OpenTelemetry Collector container image is now pinned using SHA256 digest:

image:
  repository: otel/opentelemetry-collector-contrib
  tag: 0.114.0@sha256:6ab78adda9a0b877ba55fd18e07a8c9f4d39e20ae906c9a3c79c36f37d49e7f7

This prevents tag mutation attacks where an attacker could push a malicious image with the same tag.

3. Commit SHA Documentation

Every chart version is now documented with its upstream Git commit SHA, providing a clear audit trail and enabling verification of the exact source code for each deployment.

Documentation

New: docs/version-management.md

A comprehensive 254-line guide covering:

• Current versions table with complete commit SHA references
• Detailed explanation of SHA pinning strategy and benefits
• Step-by-step procedures for updating chart versions
• How to retrieve commit SHAs and container image digests
• Troubleshooting common deployment issues
• Links to upstream documentation and resources

Updated: docs/observability-setup.md

Added a new "Version Management" section explaining:

• The charts included in the observability stack
• SHA pinning approach and security benefits
• Link to the detailed version management guide

Benefits

Security

• Supply Chain Protection: Cosign verification prevents deployment of tampered charts
• Image Immutability: SHA256 pinning eliminates tag mutation vulnerabilities
• Audit Trail: Commit SHAs provide traceable deployment history

Operations

• Reproducibility: Guarantees identical artifacts across all environments
• Safe Rollbacks: Enables confident rollback to known-good versions
• Version Tracking: Clear record of deployed versions and their sources

Compliance

• Traceable: Every deployment can be traced to source commits
• Verifiable: Cryptographic signatures prove authenticity
• Documented: Complete version history maintained in version control

Testing

• ✅ YAML syntax validated for all modified files
• ✅ Multi-document YAML structure preserved
• ✅ Documentation links and references verified
• ✅ Chart versions confirmed from upstream releases

Breaking Changes

⚠️ Grafana Major Version Upgrade (7.x → 10.x)

The Grafana upgrade is a major version change. While the configuration should remain compatible, please:

• Review Grafana 10.0 release notes
• Validate dashboards and data sources after deployment
• Monitor for any unexpected behavior

Deployment

After merging this PR:

1. Flux will automatically detect the changes
2. Cosign verification will be performed for each chart
3. Updated charts will be deployed following Flux reconciliation schedule
4. Monitor deployment: kubectl logs -n flux-system -l app=helm-controller

Rollback Plan

If issues arise, this PR can be safely reverted:

1. All versions are SHA-pinned, ensuring reliable rollback
2. Flux will automatically reconcile to previous versions
3. No manual intervention required

References

• OpenTelemetry Helm Charts
• Prometheus Community Helm Charts
• Grafana Helm Charts
• Flux Cosign Verification
• Sigstore Cosign

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.